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Abstract 

CO , 

I It is well known that the consensus problem cannot be solved de- 

• ■ terministically in an asynchronous environment, but that randomized 

Q I solutions are possible. We propose a new model, called noisy schedul- 

ing, in which an adversarial schedule is perturbed randomly, and show 
_ that in this model randomness in the environment can substitute for 

^ ■ randomness in the algorithm. In particular, we show that a simplified, 

I deterministic version of Chandra's wait-free shared-memory consensus 

algorithm (PODC, 1996, pp. 166-175) solves consensus in time at most 
I logarithmic in the number of active processes. The proof of termination 

is based on showing that a race between independent delayed renewal 
I processes produces a winner quickly. In addition, we show that the 

■ protocol finishes in constant time using quantum and priority-based 

I scheduling on a uniprocessor, suggesting that it is robust against the 

O ' choice of model over a wide range. 

^ ■ 1 Introduction 



Perhaps the single most dramatic result in the theory of distributed com- 
puting is Fischer, Lynch, and Paterson's proof of the impossibility of deter- 



ministic consensus in an asynchronous environment with failures |22]. This 
result and its extensions [^,^ show that the consensus problem, in which 
a group of processes must collectively agree on a bit, cannot be solved deter- 
ministically in an asynchronous message-passing or shared-memory model 
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if an unrestricted adversary controls scheduling. Solutions to the shared- 
memory version of this fundamental problem have thus taken the approach 
of restricting the adversary, either by allowing randomization that limits 
the adversary's knowledge ll],|,|,|l^,|l|,|l|,|l5|-|l3S by imposing timing 
constraints that limit the adversary's control |^^0|,^. As a corollary to 
granting less power to the adversary, these solutions often involve granting 
more power to the algorithm, in the form of the ability to obtain random 
bits or explicitly delay steps. By using these additional powers an algorithm 
can escape the FLP bound and reach agreement. 

These additional powers come at a cost. Randomization alone is not 
powerful enough to allow sublinear consensus protocols 0, so efficient ran- 
domized solutions have required additional constraints on the ability of the 
adversary to observe the arguments to operations and the contents of unread 
memory locations [12,13, 16|. These algorithms carefully manage common 
pools of unread random bits for future use, a clever but odd-looking prac- 
tice that is justified primarily by the specific details of the model. The 
delay-based algorithm of Q is less convoluted, but still depends on using 
explicit delays that at the minimum require that a process has the power to 
invoke them and at worst may add unnecessary delay when few processes 
participate. 

As an alternative to designing an algorithm specifically to exploit the 
weaknesses of a particular adversary model, we consider the approach of 
using a simple algorithm that guarantees agreement but relies on good luck 
to terminate. Our lean-CONSENSUS algorithm, described in Section ^, is 
obtained by removing all of the randomized parts of a similar algorithm due 
to Chandra jl^. The essential idea (which is the core of many consensus 
protocols in the literature) is to stage a race between those processes that 
prefer and those that prefer 1, with the rule that if a slow process sees 
that faster processes are all in agreement it adopts their common preference. 
The race is implemented using two arrays of atomic read/write bits. The 
algorithm terminates when the fastest processes are all in agreement and 
can decide on their preferred value safely, knowing that other processes will 
adopt the same preference before they catch up. As shown in Section ^, 
this mechanism is enough to ensure that if any one process decides then all 
other processes soon decide on the same value, no matter how the adversary 
arranges the schedule. 

In effect, the race framework allows the processes to detect agreement 
once it occurs. But unlike other consensus algorithms, lean-CONSENSUS 
makes no attempt to cajole the processes into reaching agreement — it relies 
entirely on the hope that some process eventually pulls ahead of the others. 
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In order to dash this hope, the adversary must exercise enough control to 
ensure that the fastest processes run in lockstep. We beheve that in many 
natural system models it will be difficult for the adversary to exercise this 
much control. 

One such model is what we call the noisy scheduling model, described 



in Section |3.l| . In this model, the adversary proposes a schedule that spec- 
ifies the order in which read and write operations occur, but this schedule 
is perturbed by random noise drawn from some arbitrary non-constant dis- 
tribution. This noise corresponds to random factors in a system that might 
not be strongly correlated with the algorithm's behavior, such as network 
delays, clock skew, or bus or memory contention. 

We show in Section ^ that, in the noisy scheduling model, lean-CONSENSUS 
terminates with expected 0(logn) work per process, where n is the number 
of active processes. This result is distribution-independent, in the sense that 
the algorithm's asymptotic performance does not depend on the noise dis- 
tribution in the model (though the constant factor does), and it holds even 
if processes are subject to random halting failures. Because the algorithm's 
performance depends only on the number of processes actually executing 
the protocol and not on the total number of processes in the system, it is 
adaptive in the sense of |11|, which implies it is fast in the sense of ||2|,[2^. 
Thus it is well-suited to situations where only one or a few processes attempt 
to run the algorithm at the same time. 

Our noisy scheduling model is similar to the model used by Gafni and 
Mitzenmacher [^] in their analysis of mutual exclusion protocols with ran- 
dom timing, but is extended to include constant delays inserted by the 
adversary in addition to random delays. Another source of inspiration is 
Koutsoupias and Papadimitriou's diffuse adversary [25|, which chooses a dis- 
tribution over executions in which no branch at any decision point can occur 
with probability more than some fixed e. Our model is not the first in which 
an adversary chooses parameters for a stochastic process that then con- 
trols scheduling; a sophisticated model of this type, based on asynchronous 
PRAMs, has been proposed by Cole and Zajicek p9[ |. 

To give support to our intuition that many possible restrictions on the 
adversary make lean-CONSENSUS work, we also consider what happens with 
a hybrid quantum and priority-based scheduler on a uniprocessor, following 
the approach of . (The details of this model, which subsumes both quan- 
tum scheduling and priority-based scheduling, are sketched in Section |3.2| .) 
We show in Section |^ that lean-CONSENSUS terminates in 0(1) steps in 
the hybrid-scheduling model, as long as the quantum is at least 8. The 
restriction to a uniprocessor is necessary because B shows that no deter- 
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ministic algorithm can solve consensus with multiple processors, even with 
hybrid scheduling, without using stronger primitives than atomic read/ write 
registers. 

Our basic consensus algorithm requires infinitely long arrays. Obviously 
this is undesirable in a real system. In order to bound the required space, 
we adopt a technique from and cut off the algorithm after consuming 
O(log^n) bits of space, using the preference each undecided process has at 
that point as input to a more expensive, bounded-memory consensus algo- 
rithm satisfying the validity property]^ Since the more expensive algorithm 
is only run with low probability, its higher costs do not increase the ex- 
pected time for the algorithm as a whole by more than a small constant 
factor. Details are given in Section ^. 

Section ^ describes some simulation results that show that the constant 
factors in the noisy scheduling analysis are in fact quite small for plausi- 
ble noise distributions, suggesting that the good theoretical performance 
of LEAN-CONSENSUS might actually translate into fast execution in a real 
system. 

In Section we suggest a number of directions in which the current 
work could be extended, including extensions to the noisy scheduling model. 
One interesting possibility is the inclusion of adaptive crash failures. We 
argue briefly that because lean-CONSENSUS recovers quickly from such fail- 
ures, it terminates in at most 0(/logn) work per process even if up to / 
processes fail. However, there remains an interesting open question whether 
noisy scheduling is enough to get 0(log n) performance even with G(n) crash 
failures. 

2 The Consensus Problem 

In the binary consensus problem, a group of n processes, possibly subject to 
halting failures, must agree on a bit.0 A consensus protocol is a distributed 
algorithm in which each non-faulty process starts with an input bit and 
eventually terminates by deciding on an output bit. It must satisfy the 
following three conditions with probability 1: 

^ An early example of this approach is found in the bounded-rounds randomized Byzan- 
tine agreement protocol of Goldreich and Petrank , which switches from a randomized 
to a deterministic protocol if the randomized protocol does not terminate quickly enough. 

^Some authors consider the stronger problem of id consensus, in which the decision 
value is the id of some active process. In many cases, id consensus can be solved in 
a natural way using a (IgnVdepth tree of binary consensus protocols; examples of this 
approach can be found in ]l4|l(|. 
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• Agreement. All non-faulty processes decide on the same bit. 

• Termination. All non-faulty processes finish the protocol in a finite 
number of steps. 

• Validity. If all processes start with the same input bit, all non-faulty 
processes decide on that bit.^ 

3 Model 

We assume a shared-memory system consisting of an unbounded number of 
processes that communicate only through shared atomic read/write regis- 
ters. We use the usual interleaving model, in which operations are assumed 
to occur in a sequence tti, 7r2, . . ., and in which each read operation returns 
the value of the last previous write to the same location. The order in which 
operations occur is determined by a stochastic process that is partially un- 
der the control of an adversary (Section ^]^), or directly by the adversary 
subject to certain regularity constraints (Section |3.2| ). 

3.1 Noisy Scheduling 

In the noisy scheduling model, we assume that the adversary specifies when 
operations occur (subject to an upper bound on the time between successive 
operations by the same process), but that this specification is perturbed by 
random noise. 

Formally, the adversary chooses: 

1. An arbitrary starting time Ajo for each process pi, 

2. A non-negative delay Aij between process pj's (j — l)-th and j-th 
operations, bounded by some fixed constant M, and 

3. A fixed common distribution of the random delay added to each 
type of operation vr (e.g., read or write). If process pi's j-th operation 
is of type vr, it suffers an additional delay Xij whose distribution is Ft^. 
There is no restriction on the choice of the , except that they must 

^ Some definitions of consensus replace tiie validity condition witli a weaker non- 
triviality condition tliat says tliat tliere must exist executions in wiiicfi different decision 
values occur. 
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not be concentrated on a point and must produce only non-negative 
values Xij^ 

The time of process pi's j'-th operation is given by 

j 

Sij = Ajo + ^ {\k + ^ik) ■ 
k=l 

Since we are using interleaving semantics, the effect of executing two 
operations at exactly the same time is not well-defined. To avoid ill-defined 
executions, we impose the additional technical constraint on the adversary's 
choices that the probability that any two operations occur simultaneously 
must be zero. This is automatic if, for example, the noise distributions 
are continuous. Alternatively, it can be arranged by dithering the starting 
times of each process by some small epsilon. This technical constraint does 
not qualitatively change our results. 

Below we discuss the unfairness of noisy scheduling and extensions to 
allow random failures. 

3.1.1 Unfairness 

The upper bound on the Aij and the common distribution on the Xij might 
suggest that the noisy scheduling model produces fair schedules. This is not 
entirely true for sufficiently pathological distributions. 

Theorem 1 There exists a choice of F-,^ and Aij such that for any distinct 
processes pi and pii , and any operation j, the expected number of operations 
Pif completes between pi 's j-th and {j + l)-th operations is infinite. 

Proof: Set each i<V so that X^j takes on the value 2^^ with probability 
2~'^ for A: = 1, 2, . . .. For simplicity, let us suppose that A^- = for j > 0. 
We will also assume that A and B execute no operations before time 0. 

Let X be the number of operations completed by pii between Sij and 
Sij+i- We will show that the expectation of X is infinite conditioned on the 
value of t = ISijl (the ceiling is so that we have countably many cases). 

The idea is this: for each k we have probability that Sij+i > 
Xijj^i = 2^^ . Condition on this event occurring for some particular k and 
consider how many operations pi' must execute to reach time 2'^^. Either 
(a) one of these operations takes time 2*^ or more (with probability 2^^^+^ 

^In fact, the _FV distributions can be quite bizarre; it is not required, for example, tliat 
the Xij have finite expectation. 
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per operation); or (b) a total of at least 2 faster operations, each of 
which takes at most 2^'^"^) time, must occur. If we wait only for event (a), 
we expect to see 2^~^ operations; to get the actual expected number, we 
must subtract off the expected number of operations until (a) occurs after 
(b) occurs (2^~^ again) multiplied by the probability that (b) occurs. This 
latter probability is at most (1 — ^!^)'^^~^, which goes to in the limit 
as k grows; it follows that pi' executes Q,{2^) operations on average before 
time 2^ . Of these, at most t/2 can occur before time Sij, so if /c » Igt, we 
have r2(2'^) operations on average between t and 2^ , and thus also between 
Sij and Sij+i, since Sij < t < 2^ < Sij+i. 

To get the full result, we must remove two layers of conditioning. First 
compute the expectation conditioned only on t by summing 2~^Q(2^) for 
each of the infinitely many sufficiently large k. It is not difficult to see that 
this sum diverges and the expectation is infinite. Summing over all values 
of t doesn't make it any less infinite, and we are done. I 



3.1.2 Failures 

We can extend the noisy scheduling model to allow halting failures. For each 
i and each j > let Hij = oo if process pi halts before its j-th operation 
and otherwise. Define 



S'ij — Ajo + ^ (Ajfc + Xik + Hik) , 

k=l 

with the usual convention for the extended real line that x+oo = oo+x = oo 
for any finite x. If S'^j = oo, pi's j-th operation does not occur. 

We do not include failures in the noise distributions because these 
distributions do not depend on n, and a constant probability of failure would 
mean that all processes die after O(logn) steps. Instead, we assume that 
failures occur independently with probability /i(n) per operation, where h is 
some function chosen by the adversary. The effect of stronger failure models 



is discussed in Section 10 



3.2 Quantum and Priority-Based Scheduling 

Our intuition is that LEAN-CONSENSUS should perform well in any setting 
that prevents lockstep executions. One such setting is the hybrid-scheduled 
uniprocessor model of [0], which combines the priority-based scheduling 



model of [29 1 with the quantum-based scheduling model of [Q]. In this model, 
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processes are assumed to be time-sharing a uniprocessor under the control 
of a pre-emptive scheduler. Each process has a priority, and a process may 
be pre-empted at any time by a process of higher priority. A process may 
only be pre-empted by a process of the same priority if it has exhausted 
its quantum, a minimum number of operations it must complete between 
the time it wakes up and the time at which it becomes vulnerable to pre- 
emption. There is no requirement that a process start the protocol at the 
beginning of a quantum; it may have used up some or all of its quantum 
performing other work before starting the protocol. We do not consider fail- 
ures in the hybrid-scheduling model; instead, a process may be arbitrarily 
delayed subject to the constraints on the scheduler. 

4 The LEAN-CONSENSUS Algorithm 

In this section, we describe the lean-CONSENSUS algorithm. The algorithm 
is very simple, because we are relying on randomness in the environment 
to guarantee termination and thus the algorithm itself must only guarantee 
correctness and provide the opportunity for the underlying system to quickly 
jostle it into a decision state. Structurally, it is essentially identical to the 
multi-writer register consensus protocol of Chandra [|l6| with the shared 
coins removed, leaving only the implementation from multi- writer bits of 
the "racing counters" technique that has been used in many shared-memory 
consensus protocols. It also bears some similarities to the Time-Adaptive 
Consensus algorithm of Alur et al. |^ with the delays removed. 

At each step of the algorithm, each process prefers either or 1 as its 
decision value. The conflict between the 0-preferring processes and the 1- 
preferring processes is settled by a race implemented using two arrays oq and 
ai of atomic read/write bits, each initialized to zero. Each process carries 
out a sequence of rounds, each consisting of a fixed sequence of operations. 
During round r, a process that prefers b marks location ah[r] with a one and 
looks to see if either (a) it has fallen behind its rivals who prefer (1 — 6), 
in which case it abandons its former preference and joins the winning team, 
or (b) it and its fellows have sped far enough ahead of any rival processes 
that they can safely decide b knowing that those rivals will give up and join 
the b team before they catch up. The algorithm finishes fastest when the 
pack of processes disperses quickly, so that a clear winner emerges as early 
as possible. 

Let us look more closely at the details of the algorithm. A process with 
input b sets its preference p to b and its round number r to 1. (We say that 
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a process is at round r if its round number is set to r; processes thus start 
at round 1.) It then repeatedly executes the following sequence of steps. To 
simplify the description of the algorithm, we assume that while ao and ai are 
initialized to zeroes, they are prefixed with (effectively read-only) locations 
ao[0] and ai[0], both set to 1. 

1. Read a^lr] and ai[r\. If for some 5, ah[r\ is 1 and ai_;,[r] is 0, set p to 
h. 

2. Write 1 to ap[r]. 

3. Read ai-p[r — 1]. If this value is 0, decide p and exit. 

4. Otherwise, set r to r + 1 and repeat. 

Note that in each round the process carries out exactly four operations 
in the same sequence: two reads, a write, and another read. It is tempting 
to optimize the algorithm by eliminating the write when it is already evident 
from the previous step that ap[r] is set or eliminating the last read when it 
can be deduced from the value of ai_p[r] that ai_p[r — 1] is set. However, 
this optimization reduces the work done by slow processes (whom we'd like 
to have fall still further behind) while maintaining the same per-round cost 
for fast processes (whom we'd like to have pull ahead). So we must para- 
doxically carry out operations that might appear to be superfluous in order 
to minimize the actual total cost. 

5 Agreement and Validity 

If we ignore the termination requirement, the correctness of the algorithm 
does not depend on the behavior of the scheduler. The following two lemmas 
show that the validity and agreement properties hold whenever the algorithm 
terminates. The proofs are very similar in spirit to those of Lemmas 1-4 
in 11. 

Lemma 2 No process sets ab[r] unless (a) r = 1 and b is an input value, 
or (h) r > 1 and ah[r — 1] has already been set. 

Proof: Consider the first process P that sets ab[r]. Then P does not 
read 1 from [r] at round r and does not change its preference during round 
r. If r = 1, P's preference equals its input, establishing case (a); if r > 1, P 
must have set ab[r — 1] at round r — 1, establishing case (b). I 
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Lemma 3 // every process starts with the same input bit h, every process 
decides b after executing 8 operations. 

Proof: From Lemma if no process has input 1 — 6, no process ever 
sets ai_b[l]. It follows that every process sees a zero in ai_f,[l] at round 2 
and decides b. I 

Lemma 4 // some process decides b at round r, then (a) no process ever 
writes ai_fe[r], and (b) every process decides b at or before round r + 1. 

Proof: Let P decide b at round r. We will show that this implies that 
no process ever sets ai_;,[r]. 

Suppose some process sets ai_;,[r]; let Q be the first such process. Be- 
cause Q is the first process to set ai_fe[r], it must read a from ai-f,[r] at 
the start of round r. Thus Q can only set ai_b[r] if it already prefers 1 — 6 at 
the start of round r, implying that it set ai_f,[r — 1] during round r — 1; and 
if it reads a from ai,[r] at the start of round r, preventing it from changing 
its preference after seeing a in ai_5[r]. But Q's read of a;,[r] occurs after 
Q's write to ai_f,[r — 1], which occurs after P's read of ai_;,[r — 1] at round 
r (because P reads 0), which in turn occurs after P's write to af,[r]. Thus 
Q reads 1 from ab[r], and changes its preference to b at round r. This con- 
tradicts our assumption that Q is the first to set ai_;,[r]. It follows that if 
any process decides b in round r, no process sets ai_b[r]. 

Since no process sets ai_fe[r], any process that reaches round r + 1 must 
set ab[r + 1] (by Lemma and will decide b after reading from ai_b[r]. 
Thus no process runs past round r + \ without deciding b. 

To show agreement in earlier rounds, let P' decide b' at round r' < r. 
By the preceding argument, if P' decides b' at round r', then no process sets 
ai^h'[r'] and thus (by Lemma || again) no process sets ai_fc'[r]. But since P 
sets afc[r], we must have b' = b. I 

6 Termination with Noisy Scheduling 

In this section, we show that LEAN-CONSENSUS terminates in 0(logn) rounds 
with noisy scheduling and random failures. (This analysis includes the core 
model without random failures as well, since the adversary can always choose 
h{n) = 0.) We show that either all processes die (in which case we treat the 
algorithm as terminating in the last round in which some process takes a 
step), or some group of processes with a common preference eventually gets 
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two rounds ahead of the other processes. To avoid analyzing the details of 
how processes shift preferences, we will show the even stronger result that 
unless all processes die, a single process eventually gets two rounds ahead 
of the other processes. 

To simplify the argument, we abstract away from the individual sequence 
of operations in each round and look only at the times at which rounds are 
completed. We can thus assume that the adversary provides a single noise 
distribution F (corresponding to the distribution of the sum of the delays on 
three reads and one write) and that the values Aij , Xij , and Hij provide the 
delay not on the j-th operation but on the j'-th round. Since this abstraction 
merely involves summing together the underlying variables on operations, it 
does not reduce the adversary's control over the protocol. We will scale M 
appropriately so that it is still the case that < Aij < M when j > 0. 

Using this approach, the increment Aij + Xij + Hij is the time taken 
for process i to move from the end of round j — 1 to the end round j. 
The constant Ajo represents the process's starting time, and S^y = Ajo + 
J2^j=i (Aij + Xij + Hij) gives the time at which the process finishes round r. 
A process i wins the race with a lead of c rounds at round r + c if it finishes 
round r + c before any other process finishes round r, i.e., if S'i^.^^ < S[i ^ 
for all i' ^ i. 

We would like to show a bound on how the expected round at which some 
process wins by c scales as a function of the number of processes n, keeping c, 
M, and F fixed. This bound is given in Corollary |ll| below. We will assume 
that h{n) = o(l), as otherwise all processes die after O(logn) rounds on 
average. The proof proceeds in two steps: first we show that for any r 
which some process finishes with at least constant probability, there exists 
a critical time t that gives at least a constant probability that S'i^. < t for 
exactly one i. We then show that if r is large enough, Pr[5j'^_,_^ < t|5jy < t] 
is also at least a constant. It then follows that the probability that S'i < t 
while S[,^ > t for any i' ^ i is at least the product of these two constants 
and the constant probability that pi is not killed between rounds r and r + c. 
Thus after a constant number of phases each consisting of r + c rounds we 
expect some process to win. 

6.1 Existence of a winner 

In this section, we build up the tools needed to show that for each round 
there exists a fixed time at which there is likely to be a unique winner. 
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Lemma 5 Let Ai, . . . , An be independent events. If the probability that no 
Ai occurs is x, where x is not zero, then the probability that exactly one Ai 
occurs is at least —xlnx. 

Proof: Let qi be the probability that Ai does not occur. The probabil- 
ity x that no Ai occurs is the product of the qi. Since x is nonzero, each qi 
must also be nonzero. The probability that exactly one Ai occurs is given 

by 




i=l 



n ^ 



= x —n 

\ i=i ^ / 

Let G be the geometric mean of the qi and let H be their harmonic 
mean. By the theorem of the means, G > H. Observe that G = x^/" and 

1 if Iri 2/ \ f In 3y \ 

— = ujH > n/G = nx^^'^ = nexp I J > n I 1 J = n — Inx. 

i—i Qi \ J \ ^ J 

Plugging this inequality into (|l|) gives the result. I 

Suppose Xi, . . . , Xn are random times. The following lemma shows that 
under certain conditions there exists a constant time t^, such that, with 
constant probability, at most one of the Xi is less than Iq: 

Lemma 6 Let Xi, . . . ,X„ be independent random variables such that for 
all finite values t and all distinct i,j, the probability that Xi = Xj = t is 
zero. Then either Pr[ViXj = oo] is greater than or there exists to such 
that the probability that exactly one of the Xi is less than or equal to to is 
at least 1/5. 

Proof: For each t, let qi{t) be the probability that Xi is not less than 
or equal to t. Let q{t) = qi{t) be the probability that none of the 
Xi are less than or equal to t. Note that each qi{t) is a decreasing right- 
continuous left-limited function with limf^_oo Q'i(t) = 1 and limt_+oo 9i (0 = 
Pr[Xj = oo]. Similarly, q{t) = Yliqi{t) is right-continuous, left-limited, and 
has limt^_oo Q{t) = 1 and limt^oo Qit) = Pr[VzXj = oo]. 

Suppose that this latter quantity is less than or equal to e^^. (If not, 
the first case of the lemma holds.) Then for some finite t, q{t) < e~^. Let 
to be the least such t. 
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Now suppose q{to) > e~^. Then, by Lemma ^, the probabihty that 
exactly one Xi is less than or equal to to is at least 2e~'^ ~ 0.27 . . .. 

Otherwise, we have q{to) < but q{to — ) = liuit^to- Qi't) > e^^- (We 
are using the usual convention that f{x—) denotes the left limit of / at 
X.) This discontinuity must correspond to a discontinuity in qi for some 
i. At most one qi has a discontinuity at to; by the assumption that the 
probability that distinct Xj, Xj both equal to is zero. Hence, for all j ^ i 
we have qj{tQ-) = qj{to) and thus qi{to-)/qi(to) = q{to-)/q{to) < e"^ 

Since (?i(to~) ^ 1; it follows immediately that qi{to) < and thus 
the probability that Xi is less than or equal to to is at least 1 — e~^. 
Now the probability that no other Xj is less than or equal to to is at 
least q{to)/qi{to) > q(to—) > e~^. Since the variables are independent, 
the probability that only Xi is less than or equal to to is thus at least 
(l-e~i)e-i «0.23.... I 

6.2 Size of the lead 

In this section, we show that if enough rounds have passed, a process that is 
likely to be ahead of the others is in fact likely to be several rounds ahead. 
The proof is somewhat complicated by the lack of restrictions on the noise 
distribution, but the following lemma shows how the Strong Law of Large 
Numbers can be used to smooth the noise terms out a bit. 

Lemma 7 Let Xi,X2, . . . be finite non-negative independent identically dis- 
tributed random variables whose common distribution is not concentrated 
on a point. Define Sn = J2i'=i^i- ^^r any c, there exist n,t such that 
Pi[Sn <t]<^ but Pr[5„ < t - c] > 0. 

Proof: Let us first consider the case where Xi has a finite expectation 
m. Then the Strong Law of Large Numbers says that Sn/n converges to m 
in the limit with probability 1. So for any e > 0, the probability that 5„ is 
less than m — e goes to zero and thus drops below 1/2 for all n greater than 
some no. 

Let t„ = n(m — e). As long as n > no, we have Pr[S'„ < t] < ^. Now 
suppose that Pr[5„ < t^ — c] = whenever n > no. Since the Xi are 
independent, this event can only occur if for each Xi, Xi < = m — e — ^ 
with probability 0. Taking the union of countably many such bad events 
for each rational e and each n > uq shows that the event Xi < m, also 
has probability 0. It follows that Xi > E[Xi] almost surely and thus the 
distribution of Xi is concentrated on E[Xi], a contradiction. 
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If Xi does not have a finite expectation, then Sn/n grows without bound 
with probabihty 1 (see the corollary to Theorem 22.1 in |jl^). So for any 
X, there exists no, such that Pr[5'„/n < j;] < ^ for > no- We repeat 
the above analysis for t = nx; if Pr[S'„ < t — c] = for all such t, we get 
Xi > X — almost surely, implying Xi exceeds any finite bound x. Again, 
a contradiction. I 

Once the noise terms have been smoothed, it is not hard to show that 
they eventually accumulate enough to push a winner ahead: 

Lemma 8 Fix c > 0. Let Xi, X2, . . . be finite independent identically dis- 
tributed random variables such that there exists a threshold to for which 
Fi[X < to] < I but Pt[X < to - c] = (5o > 0. Define Sn = E"=i ^i- 

Then for any e > 0, there exists an n = 0(log(l/e)), such that for any 
t, Pr[5„ < t] > e implies Pr[5„ < t — c\Sn < t] > jSq. 

Proof: Set n = 8(ln(l/e) + 1). Each Xi has probability at most 1/2 of 
being less than to, so a simple application of Chernoff bounds shows that the 
probability that 3/4 or more of the Xi are less than to is at most e~"/^ = e/e. 

We will use this fact to argue that even when conditioning on S'n < t, 
there is nearly one chance in four that X^ in particular is greater than to- 
In this case, Sn-i is less than t — to and we can use independence to replace 
Xn with a new value less than to — c, giving a sum Sn less than t — c, all 
without reducing the probability by much. 

Formally, we have the following sequence of inequalities, each of which 
is implied by the previous one. Let Pr[S'„ < t] = p and suppose p > e. Then 
we have: 





Pr[5„ < t] 




p 


Pr[5ri < t A at least \ 


of Xi are greater than to] 


> 


p-e/e 




Pr[5„ < t A X„ > to] 


> 


-{p-e/e) 




Pr[5„_i < t - to] 


> 




Pr[5„ 


-1 < t - to A X„ < to - c] 


> 


^{p-e/e)6o 




Pr[5„ < t - c] 


> 


^{p-e/e)5o 




Fl[Sn < t - c\Sn < t] 


> 


1 

-{p-e/e)6o/p 
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Since p > e, this last quantity is at least ^(1 — l/e)6o, which is in turn 
greater than j6o. I 

We can now combine Lemmas |^ and ^ into the following: 

Lemma 9 Let Xi,X2, . . . be finite non-negative independent identically dis- 
tributed random variables whose common distribution is not concentrated 
on a point. Define Sn = J2i'=i^i- -^^2; c > 0. Then there is a con- 
stant 6, such that for any e > 0, there exists n = 0(log(l/e)), such that 
Pr[5„ < t — c\Sn < t] > S whenever Pr:[Sn < t] > e. 

Proof: Use Lemma |7| to group the Xi together into partial sums 
= Z]}=°jno+i -^j '^ith the property that for some t FT[Yi < t] < ^ but 
Pr[yj < t — c] = 5o > 0. (Note that no does not depend on e, so it disappears 
into the constant factor.) Then apply Lemma |^ to sums of these Yi variables 
to get the full result. I 



6.3 When the Race Ends 

In this section, we show that a race between n independent delayed renewal 
processes with bounded added delays ends in O(logn) rounds with at least 
constant probability. In the following section, we translate this result, which 
appears as Corollary back into terms of the lean-CONSENSUS algorithm 
to get Theorem |l^. 

Theorem 10 Let {Xij}, where i,j >1, be a two-dimensional array of fi- 
nite non-negative independent identically distributed random variables with 
a common distribution function F that is not concentrated on a point. Let 
{Ajj}, where i > l,j > 0, be a two-dimensional array of constants with 
< Aij < M when j > 1. Let {Hij}, where i,j > 1, be a two-dimensional 
array of independent random variables, each of which is equal to 00 with 
probability h{n) and otherwise. Define 

r 

i=i 

Assume that for any finite t, integer r, and i ^ j, Pr[S',-^ = S'j^ = t] = 0. 
Let c be any integer constant greater than 0. 

Then there exists a constant 5 > Q, such that for any n, there exists 
r = O(logn) and t, such that 



Pr 



Vi = 00 V {^i < n : S[^j._^^ < t A^fi' ^ i,i' < n : S[i^. > t^ 



> 6. 
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The constant factor in r = O(logn) and the constant S may depend on 
c, F , M , and h; hut neither constant depends on n. 

Proof: Since each Xij is finite with probabihty 1, there exists some 
constant ci such that PrEj=r+i^«i < '^i] > ^- — J2^j=i-^ij ^^'^ 

let Sir — + X/^=o ^ir- -^Pply Lemma |9| to the sequence X^j with c — 
cM + ci and e = to obtain r = O(logn) and a constant for which 
Pi[Tir <t- cM - ci\Tir <t]>do whenever Pr[rjr < t] > n~^. Adding the 
missing constant terms X]j=o '^ir to get Sir is equivalent to subtracting 

these same terms from each occurrence of t, so we in fact have Pr[S'jr < 
t — cM — ci\Sir < t] > 60 whenever Pr[S'jr < t] > n~^. This gives us our 
target round r. 

Now apply Lemma ^ to S*,-^, for all i < n, to show that with probability 
at least 1/5 either ^iS'^^ = cxd or there exists a time to, such that there is a 
unique winner i < n for which S'^^. is less than to- Let us assume without loss 
of generality that n is at least 6. Throw out all cases where i has Pr[S'^^ < 
^o] ^ n~'^; this leaves a probability of at least 1/5 — 1/n > 1/30 that (a) there 
is a unique winner i, and (b) i satisfies the condition Pr[S'jY < to] > n~'^, 
implying Pr[5ir = S'^^. < to] > and thus Pr[S'jj. < to — cM — ci\Sir < 
to] > So- So with probability at least ^Sq, we have Sir < to — cM — ci, 
and thus with probability at least we have Si^r+c < Sir + cM + ci = 
S'ij. + cM + ci < to. 

Suppose that this event holds. It is still possible for S^ to be infinite 
if E}=r+i Hij = 00. Call this event /; if Pr[/] = 1 - (1 - h{n)y > j^6o, 
then h{n) is bounded below by a constant and there exists r' = O(logn) 
such that Pr[VzSjy/ = 00] is at least a constant. Alternatively, we have 
Pr[5j' = Si^r+c < S[,^. + cM + ci\> 5 = -^^o- In either case, the theorem 
holds. I 

Corollary 11 Let R he the first round for which either 

• There exists i, such that S[ jij^^ < S^,j^ for all i' 7^ i, or 

• For all i, S[ j^^^ = 00. 

Under the conditions of the preceding theorem, E[R] = O(logn), and, for 
any k>0, Fi[R >k]< e-L'=/o(i°g")J. 

Proof: Theorem |l^ says that the desired event occurs with constant 
probability 6 after a phase consisting of r = O(logn) rounds. If it does not 
occur, we can apply the theorem again to the subset of the i's for which 
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S[ is finite, starting with round r + c + 1 and setting the initial delay Ajo 
to the value of from the previous phase. 

On average, at most 1/5 = 0(1) such phases are needed, giving E[-R] < 
(l/5)r = O(logn). For the exponential tail bound, observe that the proba- 
bility that the algorithm runs for more than c phases of r rounds each is at 

most (1 - 6y = ((1 - 6)^/^y^ < {e-^y^ = e"^''. So the probability that the 
algorithm runs for more than k rounds is at most e~^''^^^^ < e^Lfc/0(iogn)J ^ | 

6.4 When lean-consensus Ends 

Translating Corollary |ll| back into terms of the lean-CONSENSUS algorithm 
gives: 

Theorem 12 Under the noisy scheduling model with random failures, start- 
ing from any reachable state in the lean-CONSENSUS algorithm in which the 
largest round number of any process is r, the algorithm running with n active 
processes terminates by round r + r', where r' has expected value O(logn) 
and Pr[r' > k] < e'l'^/Oiiogn)] ^ > q. 

Proof: Apply Corollary |l^ with c = 2 and the initial delays A^o set to 
the times at which each process completes round r. This shows that after 
R additional rounds, where E[^] = O(logn) and Ft[R > k] < g-L'^/oCiogn)] ^ 
either some process P finishes some round s before any other process finishes 
round s — 2, or all processes fail. In the first case, if P prefers b, it is the 
only process to have written to ab[s — 1] or ai-b[s — 1] by the time it reads 
ai_fe[s — 1] as part of round s. Thus it reads a zero from ai_b[s — 1] and 
decides. All other processes decide at most one round later by Lemma ^ We 
thus get r' < R+1, and the single extra round disappears into the constant 
factors. I 

It is not hard to see that an O(logn) bound is the best possible, up to 
constant factors. 

Theorem 13 There exists a noise distribution F and a set of delays A such 
that the LEAN-CONSENSUS algorithm requires expected r2(logn) rounds in the 
noisy scheduling model, even without failures. 

Proof: Let all Ajj = for j > 0, and let F have each operation take 
either 1 or 2 time units with equal probability. Then any single processor 
completes its first logn operations in 1 time unit each with probability 1/n. 
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To avoid simultaneous operations, let Ajo be some small distinct epsilon 
value for each i. 

Start n/2 processes with input and n/2 with input 1. The probability 
that there exists at least one 0-input process and at least one 1-input process 
that both complete their first logn operations in 1 time unit each is given 
by 




which goes to (1 — e~^/^)^ = in the limit as n grows. So there is 

a constant probability that at least one process with each input runs for 
logn operations without ever changing its preference to that of a faster 
process with the opposite preference, and we get expected il(logn) rounds 
of disagreement. I 



7 Termination with Quantum and Priority-Based 
Scheduling 

In this section, we consider the question of termination subject to hybrid 
quantum and priority-based scheduling on a uniprocessor. The required 
quantum size is 8 operations; curiously, this is the same size required for 
the specialized algorithm given in We see this coincidence as hinting at 
the possibility that all shared-memory consensus algorithms may ultimately 
converge to a single ideal algorithm (though such an ideal algorithm, if it 
exists, is probably not identical to lean-CONSENSUS). 

Theorem 14 When running LEAN-CONSENSUS in a hybrid- scheduled sys- 
tem with a quantum of at least 8 operations, every process decides after 
executing at most 12 operations. 

Proof: We will show that at most one of ao[l] and ai[l] is set before 
some process finishes round 2 and decides. Consider an execution in which 
ao [1] and oi [1] are each set at some point. Let Pq and Pi be the first processes 
to set ao[l] and ai[l], respectively. Neither Pq nor Pi can have observed the 
round-1 write of the other, or it would have changed its preference. Thus 
both processes' round-1 reads of ao[l] and ai[l] must have occurred before 
either performed its round-1 write. Since we are on a uniprocessor, this can 
only occur if one of the processes was pre-empted before its write occurred. 
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Assume without loss of generality that Pq is this unlucky process. Since 
Pq is the first process to write to aQ[l], if we can show that Pq is not resched- 
uled before some process completes round 2, then that process decides 1 (and 
by Lemma 1^, all processes eventually decide 1) as soon as it observes a zero 
in ao[l]- So we need only show that Pq is not rescheduled until some other 
process completes eight operations. 

Let Qi be the process that pre-empts Pq. At the time of pre-emption, 
Qi is at the start of a quantum; it either finishes eight operations without 
being pre-empted or is pre-empted by a higher-priority process Q2- But 
Q2 in turn can only be pre-empted before completing its quantum by some 
higher-priority process Q3. After at most n such pre-emptions, we run out of 
higher-priority processes, and the last process runs to the end of its quantum 
and decides. Note that all of the processes in this chain (except possibly Qi) 
have a higher priority than Pq and thus cannot be equal to Pq. It follows 
that some process finishes round 2 before Pq is rescheduled, and thus every 
process decides 1 by the end of round 3. I 

8 Bounded space LEAN-CONSENSUS 

The LEAN-CONSENSUS algorithm as described in Section ^ requires infinite 
space. In this section, we describe how to modify the algorithm to use 
bounded space. We assume that we have available a backup protocol, which 
is a bounded-space consensus protocol that requires polynomial work per 
process (for example, the 0(n'^) protocol in Q works). We will build a 
protocol that combines lean-CONSENSUS with the backup protocol in a 
way that only uses the backup protocol rarely, so that its high cost adds 
only a constant to the O(logn) cost of the combined protocol. 

Note that such a combined protocol is not necessary in the model of Sec- 
tion]^ as in that model we only need space for 3 rounds of lean-CONSENSUS. 

The combined protocol operates as follows: 

1. Run LEAN-CONSENSUS through round rmax- 

2. At round r^ax + Ij switch to the backup protocol, using the preference 
at the end of round rmax of lean-CONSENSUS as input to the backup 
protocol. 

If ''max is large enough, most of the time we will expect that lean- 
CONSENSUS terminates before reaching rmax and the backup algorithm will 
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not be used. But in the case where rmax is reached (say, because the sched- 
uler is nastier than we have assumed), the backup algorithm guarantees 
termination using bounded space and bounded (but possibly very large) 
expected time. 

Theorem 15 For any polynomial-work consensus protocol chosen as a backup 
algorithm and any noise distribution, there is a choice of Tmax = 

0(log2 n) 

such that the combined algorithm described above is a consensus protocol 
that requires O(logn) expected operations per process and O(log^n) bits in 
the oq and ai arrays. 

Proof: First let us show that the combined algorithm solves consensus. 
Validity is immediate from Lemma ^; when all inputs are equal, we never 
get past round 2 and the combined algorithm behaves identically to lean- 
CONSENSUS. For agreement, the only tricky case is when some processes 
decide during lean-CONSENSUS and others decide during the backup proto- 
col. But if some process P decides b at or before round r, then by Lemmas § 
and ^ no process writes ai_;,[r] and every process that executes the backup 
protocol has b as input. Thus the validity condition for the backup protocol 
implies that all processes decide b. 

Now let us show that there is a choice of rmax that gives the desired 
performance bound. Suppose each process finishes the backup protocol in 
0{n^) expected operations. By Theorem there is a value T = O(logn) 
such that the probability that lean-CONSENSUS does not finish by round k is 
at most e~ ^^/'^^ . Setting rmax = T-c- log n = O (log^ n) , the backup protocol 
is run with probability at most e"*^^"^" = n~'^, and thus it contributes at 
most n~'^0{rf') = 0(1) to the expected cost. 

Finally, the size of the oq and ai arrays is clearly equal to rmax = 
0{\og^n). I 

9 Simulation Results 

Figure |^ gives the results of simulating LEAN-CONSENSUS with various in- 
terarrival distributions. These simulations are of the model as described in 
Section |3.1| ; in particular it is assumed that all operations take zero time 
and that there are no contention effects or synchronization issues. 

The X axis is plotted on a logarithmic scale and represents the number 
of processes. The Y axis is plotted on a linear scale and represents the round 
at which the first process terminates (which may be one less than the round 
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Figure 1: Results of simulating LEAN-CONSENSUS with various interarrival 
distributions. 

at which the last process terminates). Each point in the graph represents an 
average termination round in 10,000 trials with the given distribution and 
number of processes. The starting times for all processes are the same except 
for a small random epsilon, generated uniformly in the range (0, 10~^). In 
each case, half the processes are started with input and half with input 1. 
There are no failures. 

The random number generator used was drand48. The distributions 
used were: 

1. Normal distribution with mean 1 and standard deviation 0.2 (variance 
0.04), rejecting points outside (0,2). 

2. 2/3 or 4/3 with equal probability. 

3. 0.5 plus an exponential random variable with mean 0.5. This corre- 
sponds to a delayed Poisson process. 

4. Geometric with p = 0.5. 



5. Uniform in (0,2). 
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6. Exponential with mean 1. This corresponds to a Poisson process with 
no initial delay; it is also equivalent to generating a schedule by choos- 
ing one process uniformly at random for each time unit. 

It is worth noting that while the expected number of rounds grows log- 
arithmically for most distributions, both the rate of growth and the initial 
value are small. These small constant factors may be the result of most 
processes adopting the values of early leaders, so that termination can be 
reached by agreement among leaders rather than the emergence of a single 
leader. 

The inverted behavior with a normal distribution is intriguing; it sug- 
gests that with large numbers of processes there are more chances for one 
particularly speedy process to leap ahead of its competitors, and that for 
some distributions this effect overshadows the effect of having more com- 
petitors to leap ahead of. It is not clear from the data whether this curve 
eventually turns around and starts rising again, or whether it converges to 
some constant asymptote. 

10 Conclusions, Extensions, and Future Work 

We see this paper as making two main contributions. The first is the ex- 
traction of the adaptive 0(logn) time LEAN-CONSENSUS protocol from its 
more sophisticated predecessors and the demonstration that this simplified 
algorithm can solve consensus in models that are less extreme than those 
predecessors were designed to survive but that are perhaps closer to captur- 
ing the scheduling behavior an algorithm is likely to experience in practice. 
Although LEAN-CONSENSUS does not really contain any new ideas, we be- 
lieve that ripping out features that practitioners might balk at implementing 
is a valuable task in its own right. 

The second is the noisy scheduling model. This model limits the adver- 
sary not by covering its eyes but by making its hands shake. It allows us 
to express the understanding that in the real world failures and timing are 
usually not fully under the control of intelligent demons, while still retain- 
ing a healthy respect for the subtlety and unpredictability of the world. We 
believe that this "perturbed worst-case analysis" approach is likely to have 
applications in many areas both in and outside of distributed computing. 

There are still many questions left unanswered and many ways in which 
the noisy scheduling model could be extended. We discuss some of these 
issues below. 
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Non-random failures. It would be nice to understand how lean-CONSENSUS 
fares with failures that are not random. We can get an upper bound in this 
situation by restarting Theorem ^ whenever a process dies. Since the ad- 
versary must kin at least one process every expected O(logn) rounds, the 
algorithm terminates in expected 0{f log n) rounds where / is the number of 
failures. This bound compares favorably with the 0(nlog^ n) work per pro- 
cessor needed by the best known randomized algorithm that solves consensus 
with a fully-adaptive adversary and up to n — 1 failures |Q, but the fully- 
adaptive adversary is much stronger than one limited to noisy scheduling. 
It seems likely that a better upper bound than 0(/logn) could be obtained 
by a more careful analysis that includes how processes change preferences. 
We conjecture that the real bound is in fact O(logn). 

Statistical adversaries. We would also like to do away with the fixed 
bound M on the delay between operations under the control of the adver- 
sary. The technical reason for including this bound in the model is that 
it provides a scale for the noise introduced by the Xij variables; if the ad- 
versary can increase without limit, it can construct a steadily slower 
and slower execution in which the noise, relative to the gap between rounds, 
never accumulates enough to affect the schedule. But a weaker statistical 
constraint, such as requiring J2^j=i ^ij — fM, might avoid such Zeno-like 
pathologies while allowing more variation in the gaps between operations.^ 
The present proof does not work with just this statistical constraint (the 
particular step that breaks down is the use of Lemma ^ to show that being 
ahead at round r often means being ahead by c at round r), but we conjec- 
ture that the statistical constraint is in fact enough to get termination in 
O(logn) rounds. 

Synchronization and contention. Though the present work was moti- 
vated by a desire to move away from powerful theoretical adversaries toward 
a model more closely reflecting the non-maliciousness of misbehavior in real 
systems, we cannot claim that the model accurately describes the behavior of 
any real shared-memory system. One difficulty is that real shared-memory 
systems generally do not guarantee full serializability of memory operations 
in the absence of additional synchronization operations (see Section 
8.6]). We can overcome this difficulty by adding synchronization barriers to 
each round of lean-CONSENSUS; in principle this does not affect the analysis 
since the structure of each round is still the same as all other rounds. A 

^This is a bit like using the statistical adversary of |lq|. 
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second problem is memory contention, which we have not analyzed. The 
difficulty with both explicit synchronization and memory contention is that 
their effects are unlikely to be consistent with the assumption that the tim- 
ing of different processes' operations are independent. To the extent that 
this lack of independence disperses processes (say, by slowing down laggards 
fighting over congested early-round registers while allowing the speedy to 
sail through relatively clear late-round registers), it helps the algorithm. 
Whether such an effect would occur in practice cannot easily be predicted 
without experimentation. 

Lower bounds. The noisy scheduling model is friendly enough that an 
O(logn) running time for consensus might not be the best possible. A 
counterexample like the one given in the proof of Theorem might be 
able to show that no deterministic algorithm with certain strong symmetry 
properties (such as no dependence on process identity and a mirror-image 
handling of the different inputs) can do better, but it not obvious where to 
look for a more general lower bound. It is not out of the question that a 
clever algorithm could solve consensus with noisy scheduling in as little as 
0(1) time. 

Message passing. All of our results are set in a shared-memory model. 
It would be interesting to see whether a noisy scheduling assumption can be 
used to solve consensus quickly in an asynchronous message-passing model. 

Other problems. Finally, though we have concentrated on a particularly 
simplified protocol for solving a single fundamental problem, it would be 
interesting to see how other algorithms fare in the noisy scheduling model. 
It seems likely, for example, that algorithms designed for unknown-delay 
models such as Alur et al.'s Q should continue to work in the noisy schedul- 
ing model, perhaps with some constraint on the noise distribution to ex- 
clude random delays with unbounded expectations. Similarly the line of 
inquiry started by Gafni and Mitzenmacher ||2^, on analyzing the behavior 
of timing-based algorithms for mutual exclusion and related problems with 
random scheduling, could naturally extend to the more general model of 
noisy scheduling. 
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